Exposure control based on image sensor cost function

ABSTRACT

The present invention relates to a method, a computer-readable medium, a computer program and an apparatus for exposure control. A histogram of the number of image sensor area elements of an image sensor that receive light at specific light reception rates from an image target is determined. At least one exposure time is determined for capture of said image target based on said histogram and on a cost function that expresses a performance of said image sensor as a function of light reception rate per image sensor area element and exposure time.

FIELD OF THE INVENTION

This invention relates to a method, a computer-readable medium, acomputer program and an apparatus for exposure control.

BACKGROUND OF THE INVENTION

Exposure control is, inter alia, required in the context of digitalphotography, where a target image is captured through a lens by using animage sensor, such as for instance a Charge Coupled Device (CCD) orComplementary Metal-Oxide Semiconductor (CMOS) image sensor. Such imagesensors are for instance installed in digital cameras such as DigitalStill Cameras (DSC) or in electronic devices that offer a camerafunction, such as for instance a mobile phone. Therein, exposure controldetermines the exposure time during which the target image is capturedby the image sensor.

Exposure control is particularly demanding if the dynamic range of animage sensor is limited. Therein, the dynamic range of an image sensormay be understood as the ratio between high and low extremes in a set ofintensity values that can be captured by the image sensor. In case oflimited dynamic range, the problem arises that details in the brightarea have to be traded against details in the dark area. With a largeexposure time, the pixels of the image sensor associated with brightparts of the captured image will be driven into saturation, so thatthere are no details in the bright part, whereas with a small exposuretime, the pixels of the image sensor associated with the dark parts ofthe captured image will only receive small or none activation at all, sothat there are no details in the dark part.

To avoid image degradation due to limited dynamic range, High DynamicRange (HDR) cameras have been proposed, providing dynamic ranges of12-20 bits per color component compared to the 6-12 bits per colorcomponent offered by normal cameras.

A first type of HDR cameras achieves increased dynamic range byproviding an image sensor with high and low sensitivity pixels in thesensor.

A second type of HDR cameras extends the dynamic range by merging imagedata of two or more images that have been captured with differentexposure times (e.g. one short exposure time for capturing details inthe bright part of the image and one long exposure time for capturingdetails in the dark part of the image). In this way, a singlehigh-dynamic range image can be created. It is readily understood thatthe two or more images have to be captured within a short time period toavoid generation of motion artifacts. In this second type of HDRcameras, properly controlling the exposure time used for the capture ofthe single images that are to be merged into a high-dynamic range imageis of crucial importance for the quality of the merged high-dynamicrange image.

SUMMARY

It is, inter alia, an object of the present invention to provide amethod, computer program, computer program product and apparatus forexposure control.

According to a first aspect of the present invention, a method isdisclosed, comprising determining a histogram of the number of imagesensor area elements of an image sensor that receive light at specificlight reception rates from an image target; and determining at least oneexposure time for capture of the image target based on the histogram andon a cost function that expresses a performance of the image sensor as afunction of light reception rate per image sensor area element andexposure time.

According to a second aspect of the present invention, a computerprogram is disclosed, comprising instructions operable to cause aprocessor to determine a histogram of the number of image sensor areaelements of an image sensor that receive light at specific lightreception rates from an image target; and instructions operable to causea processor to determine at least one exposure time for capture of theimage target based on the histogram and on a cost function thatexpresses a performance of the image sensor as a function of lightreception rate per image sensor area element and exposure time. Thecomputer program may for instance be stored in the memory of a processorof a digital camera or an electronic device that is equipped with adigital camera and may be executed by the processor.

According to a third aspect of the present invention, acomputer-readable medium having a computer program stored thereon isdisclosed, the computer program comprising instructions operable tocause a processor to determine a histogram of the number of image sensorarea elements of an image sensor that receive light at specific lightreception rates from an image target; and instructions operable to causea processor to determine at least one exposure time for capture of theimage target based on the histogram and on a cost function thatexpresses a performance of the image sensor as a function of lightreception rate per image sensor area element and exposure time. Thecomputer-readable medium may for instance be an optic, electric,magnetic or electro-magnetic storage medium, which may be fixedlyattached to or removable from an apparatus that contains the processorthat executes the computer program stored on the computer-readablemedium.

According to a fourth aspect of the present invention, an apparatus isdisclosed, comprising a processing unit configured to determine ahistogram of the number of image sensor area elements of an image sensorthat receive light at specific light reception rates from an imagetarget; and to determine at least one exposure time for capture of theimage target based on the histogram and on a cost function thatexpresses a performance of the image sensor as a function of lightreception rate per image sensor area element and exposure time.

The present invention further relates to an apparatus, comprising meansfor determining a histogram of the number of image sensor area elementsof an image sensor that receive light at specific light reception ratesfrom an image target; and means for determining at least one exposuretime for capture of the image target based on the histogram and on acost function that expresses a performance of the image sensor as afunction of light reception rate per image sensor area element andexposure time.

The apparatuses according to the present invention may for instance be adigital camera, or an electronic device that is equipped with a digitalcamera. Equally well, said apparatus may be a camera module, an externalaccelerator, an imaging engine, an application processor, or a basebandprocessor, to name but a few possibilities.

According to the present invention, a histogram is determined. Eachordinate value of this histogram, which is associated with a specificlight reception rate as abscissa value, indicates the number of imagesensor area elements of an image sensor that receive light at thisspecific light reception rate from an image target. Therein, the imagesensor may for instance be a CCD or CMOS image sensor, and the imagesensor area elements may for instance be pixels or groups of pixels ofthe image sensor. The image sensor may for instance be contained in adigital camera or in an electronic device that is equipped with adigital camera, such as for instance a mobile phone. The light receptionrate expresses at which rate light is received by the image sensor areaelements. Therein, the light reception rate may relate to single colorcomponent of the received light, or to all color components. The lightreception rate may for instance be measured in photons per time unit, orin electrons per time unit, reflecting that the image sensor areaelements output electrons in response to the reception of photons asdefined by the quantum efficiency of the image sensor area elements.

The histogram thus may provide an impression which light reception ratesare experienced by a large number of pixels of the image sensor andwhich are not.

The determining of the histogram may for instance be based on image datagathered prior to actual image capture of the image target, for instancefrom a viewfinder image, or on image data gathered at least partiallyduring actual image capture.

During or after the determining of the histogram, at least one exposuretime for image capture of the image target is determined based on thehistogram and on a cost function. Therein, the cost function expressesthe performance of the image sensor as a function of light receptionrate and exposure time. For a specific exposure time, the cost functionthen degenerates to a function of the light reception rate only.Therein, the cost function may of course depend on further parameters,as for instance an analog gain of the image sensor. As with thehistogram, also the light reception rate, which, for a specific exposuretime, serves as an abscissa for the cost function, may for instancerelate to a single color component or to all color components, and mayfor instance be expressed in terms of photons per time unit or electronsper time unit. Representations of the cost function, for instance fordifferent exposure times, may be fixedly stored in a memory, but mayequally well also be determined anew in certain intervals, for instanceif image sensor parameters have changed. The cost function may, interalia, consider noise characteristics and/or nonlinearities of the imagesensor.

Determination of the at least one exposure time thus may exploit thatthe histogram reveals how many image sensor area elements receive lightat small, medium or large light reception rates, and that the costfunction indicates for which light reception rate or rates and for whichexposure time the image sensor has optimum performance. By combininginformation on both the histogram and the cost function, thus a suitedexposure time can be determined.

According to a first aspect of the present invention, the determining ofthe at least one exposure time is based on the histogram and on aplurality of cost function representations, wherein each of the costfunction representations expresses a performance of the image sensor asa function of light reception rate per image sensor area element for aspecific exposure time. Equally well, the determining of the at leastone exposure time may be based on the histogram and only one costfunction representation. The one or more cost function representationsmay be available prior to the determining of the at least one exposuretime, or may be determined or generated during said determining of theat least one exposure time, for instance based on a mathematical modelof the cost function and/or on already available cost functionrepresentations.

Therein, at least one of the cost function representations may forinstance be based on measurements, or on an analytical model of the costfunction. Equally well, at least one of the cost functionrepresentations may be obtained from interpolation or extrapolation ofother cost function representations.

According to a second exemplary embodiment of the present invention,which is based on the first exemplary embodiment of the presentinvention, only one exposure time is determined, and the one exposuretime is determined by selecting, based on the histogram, a cost functionrepresentation out of the plurality of cost function representations andby determining the specific exposure time of the selected cost functionrepresentation as the one exposure time.

In this selection of the cost function representation (and theassociated exposure time), the determined histogram is considered, sothat an exposure time may be selected that is associated with a costfunction representation which has a good performance in one or morespecific ranges of light reception rates that are experienced by a largenumber of pixels in the image sensor when the image is captured. Forinstance, if the histogram reveals that a large number of pixels receivelight at large light reception rates, a cost function representation maybe selected that indicates high performance of the image sensor forlarge light reception rates, wherein this cost function representationmay for instance be a cost function representation for a small exposuretime.

In the second exemplary embodiment of the present invention, theselecting of the cost function representation may comprise: for eachcost function representation in the set of cost functionrepresentations, multiplying, for a range of light reception rates, therespectively associated histogram value and the respectively associatedvalue of the cost function representation and summing up the resultingmultiplication products to obtain a sum value; comparing the sum valuesof all cost function representations in the set of cost functionrepresentations to identify the largest sum value, and selecting thecost function representation that produced the largest sum value. Inthis way, the exposure time is determined in a way that an optimal matchof the performance of the image sensor to the exposure requirements ofthe image target is achieved.

According to a third exemplary embodiment of the present invention,first and second exposure times for respective first and second capturesof the image target are determined. Determining two or even moreexposure times is for instance advantageous if multiple captures of thesame image target shall be performed to increase the dynamic range.

In this third exemplary embodiment, one of the first and second exposuretimes may be determined, based on the histogram and on the costfunction, as the largest possible exposure time that still does notcause pixel saturation. This may for instance be the smaller exposuretime of the first and second exposure time.

According to a fourth exemplary embodiment of the present invention, thethird exemplary embodiment of the present invention further comprises:capturing the image target with the first exposure time to obtain afirst image; capturing the image target with the second exposure time toobtain a second image, and merging image data of the first and secondimages into a third image. By capturing the first and second image withdifferent exposure times and merging the image data of the resultingfirst and second images into image data of the third image, the dynamicrange—compared to the single image capture case—can be extended. Forinstance, if the first image is captured with a small exposure time, andthe second image is captured with a large exposure time, the first imagemay be likely to show few details in dark image portions and moredetails in bright image portions, whereas the second image may be likelyto show more details in the dark portions and few details in the brightimage portions (due to saturation), so that combining image datarelating to the bright image portions of the first image and image datarelating to the dark portions of the second image yields a third imagewith extended dynamic range. Thus an increase in dynamic range isachieved without requiring an increase in the bit depth of ananalog-to-digital converter that converts the output signals of theimage sensor into digital values for further processing.

In the fourth exemplary embodiment of the present invention, thecapturing with the first exposure time and the capturing with the secondexposure time may be temporally non-overlapping or temporallyoverlapping. The temporally non-overlapping capture may for instance beachieved by subsequent image captures. The temporally overlappingcapture may for instance be achieved by using two images sensors, or byusing only one image sensor for the capture of both images. The imagewith the larger exposure time may then for instance be derived from theimage with the smaller exposure time, for instance by means ofmultiplication with one or more factors. It is also possible that twoexposures are captured partially at the same time. The values over somethreshold (thr, if this value is digitized) are clipped after a first(longer) exposure time (t1), and then the exposure with a second(shorter) exposure time (t2) is immediately continued. The pixels afterthese two exposures are AD converted, and so the digitized pixel values(x) are produced using the known nonlinear sensor response that can belinearized (y) afterwards. The nonlinear pixel response may then forinstance be linearized by using the following equations:

If (x<thr*(t1+t2)/t1) then y=x

else y=(x−thr)*(t1+t2)/t2

In the fourth exemplary embodiment of the present invention, the mergingof the image data of the first and second image may comprise: in case ofimage data of the second image falling below a pre-defined threshold,using the image data of the second image as image data for the thirdimage; and in case of image data of the second image equaling orexceeding the pre-defined threshold, using a scaled representation ofimage data of the first image as image data for the third image, whereinthe ratio between the second exposure time and the first exposure timeis used as scaling factor.

In the fourth exemplary embodiment of the present invention, the mergingof the image data of the first and second image may alternativelycomprise: in case of a difference between image data of the second imageand a scaled representation of image data of the first image fallingbelow a pre-defined threshold, using the image data of the second imageas image data for the third image, wherein the ratio between the secondexposure time and the first exposure time is used as scaling factor; andin case of the difference between the image data of the second image andthe scaled representation of the image data of the first image equalingor exceeding the pre-defined threshold, using the maximum value of theimage data of the second image and the scaled representation of theimage data of the first image as image data for the third image.

According to a fifth exemplary embodiment of the present invention,first and second exposure times for respective first and second capturesof the image target are determined by selecting, based on the histogram,first and second cost function representations out of the plurality ofcost function representations and by determining the specific exposuretimes of the selected first and second cost function representations asthe first and second exposure times.

In this fifth exemplary embodiment of the present invention, theselecting of the first and second cost function representations maycomprise: forming a plurality of different combinations of two costfunction representations out of the plurality of cost functionrepresentations; determining, for each combination of cost functionrepresentations, a sum value by applying a maximum-value function to thetwo cost function representations to obtain a maximum-value costfunction representation, by multiplying, for a range of light receptionrates, the respectively associated histogram value and the respectivelyassociated maximum-value cost function representation value and bysumming the resulting products; and selecting the two cost functionrepresentations of that combination of cost function representationsthat yields the largest sum value. Therein, the maximum-value functionselects, for each abscissa value of the two cost functionrepresentations, the larger of the two respectively associated costfunction representation values.

A sixth exemplary embodiment of the present invention further comprisesdetecting motion of at least one of the image target and the imagesensor; and weighting the cost function representations in dependence onthe detected motion prior to or during the determining of the at leastone exposure time. In this way, the cost function representations alsoreflect the impact of motion during image capture. It may also bepossible that the cost function representations are weighted with motionwithout actual motion detection (e.g. if a sport mode is selected).

In this sixth exemplary embodiment of the present invention, theweighting of the cost function representations may depend on thespecific exposure times of the cost function representations. Forinstance, cost function representations associated with larger exposuretimes may be weighted by multiplication with smaller factors than costfunction representations associated with shorter exposure times.

According to a seventh exemplary embodiment of the present invention,the performance of the image sensor is related to a signal-to-noiseratio of the image sensor.

According to an eighth exemplary embodiment of the present invention,the cost function expresses a performance of the image sensor as afunction of light reception rate per image sensor area element, exposuretime and analog gain of the image sensor, and wherein at least oneexposure time and at least one analog gain for capture of the imagetarget are determined based on the histogram and on the cost function.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE FIGURES

In the figures show:

FIG. 1: a schematic block diagram of an exemplary embodiment of anapparatus according to the present invention;

FIG. 2: a flowchart of an exemplary embodiment of a method according tothe present invention;

FIG. 3: a flowchart of a further exemplary embodiment of a methodaccording to the present invention;

FIG. 4 a: an example of a histogram for an image target and of threecost function representations according to the present invention; and

FIG. 4 b: a further example of a histogram for an image target and ofthree cost function representations according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 depicts a schematic block diagram of an exemplary embodiment ofan apparatus 1 according to the present invention. Apparatus 1 comprisesa camera unit 16 for capturing images, wherein the exposure time for theimage capture is controlled according to the present invention.

Camera unit 16 comprises an image sensor 160, such as for instance aCharge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor(CMOS) image sensor, which is configured to capture images projectedonto its surface via according camera optics. Image sensor 160 may beequipped with an analog to digital converter for transforming signalsrepresenting a captured image into digital data.

Camera unit 16 can further comprises a motion detector 161 for detectingmotion of the image target and/or apparatus 1. Motion detector 161 mayfor instance be embodied as motion sensor. Equally well, motiondetection may be performed by central processor 10 based on the analysisof differences between subsequently captured images.

Camera unit 16 is furthermore equipped with a shutter unit 162, whichcontrols the opening of a shutter according to a prescribed exposuretime. During this exposure time, images are projected onto the surfaceof image sensor 160. Therein, the shutter can be implemented eithermechanically or electronically. Furthermore, the shutter may beimplemented either using rolling shutter or global shutter.

It is understood by those skilled in the art that camera unit 16 maycomprise further functional units such as a flash unit for controlling aflash operation, and an auto focus unit for controlling an auto focusoperation of camera unit 16, to name but a few possibilities.

Apparatus 1 further comprises a central processor 10 for controlling theoverall operation of apparatus 1. In particular, central processor 1 isconfigured to determine one or more exposure times according to thepresent invention, as will be discussed with reference to the flowchartsof FIGS. 2 and 3 below, and to control image sensor 160, motion detector161 and shutter unit 162 to allow capturing of one or more images withthese one or more determined exposure times. In case that more than oneimages for a specific image target are captured with different exposuretimes, central processor 10 is further configured to merge the imagedata of these images into a resulting image with extended dynamic range.It should be noted that merging may also be performed in image sensor160. This may require that there is a memory in image sensor 160.Alternatively, the long and short exposures for one pixel may be appliedso that only a small buffer is needed or that no buffer is needed atall. It may also be possible to compress the captured dynamic of thepixel by using some compression method or by using some nonlinearfunction.

Apparatus 1 further comprises a display 11, a user interface 15 and animage memory 13 for storing captured images. Image memory 13 may forinstance be a removable memory, such as for instance a memory stick orcard. Display 11, user interface 15 and image memory 13 are allcontrolled by central processor 10.

Central processor 10 may run program code stored in processor memory 12,which may for instance be embodied as Random Access Memory (RAM),Read-Only-Memory (ROM), to name but a few possibilities. Processormemory 12 may equally well be embodied as a memory that is removablefrom apparatus 1. The program code stored in processor memory 12 may forinstance define the way how central processor 10 controls the units ofapparatus 1, and may particularly define how exposure control isperformed by exposure control unit 110 of central processor 10 accordingto the present invention.

Apparatus 1 may for instance represent a digital camera, where display11 then may function as a viewfinder and as a means for displayingcaptured images, and user interface 15 may comprise interaction elementssuch as a camera trigger, control elements for zooming and controlelements for operating a menu structure. Therein, display 11 may also atleast partially function as user interface, for instance by displaying amenu structure. Display 11 may also be embodied as a touch keypad andthus also function as a user interface.

Equally well, apparatus 1 may represent an electronic device that isadditionally furnished with functionality to capture images. Forinstance, apparatus 1 may represent a mobile appliance such as a mobilephone, a personal digital assistant or a laptop computer. Therein,central processor 10 may then for instance be the standard processor forcontrolling the functioning of the mobile appliance, display 11 may beits standard display, and user interface 15 its standard user interface,such as for instance a keyboard or keypad. Similarly, memories 12 and 13may be standard components already contained in the mobile appliance. Inorder to furnish the mobile appliance with the functionality to captureimages, camera unit 16 may be added to the mobile appliance, and theprogram code in processor memory 12 may be accordingly altered to enableprocessor 10 to control camera unit 16 to capture images under usage ofthe exposure control according to the present invention.

Moreover, FIG. 1 illustrates that apparatus 1 may further comprise adedicated exposure control unit 14, which may partially or entirelyimplement exposure control according to the present invention and thusmay reduce the computational burden imposed on central processor 10.Dedicated exposure control unit 14 is however optional and thus depictedin dashed lines. Dedicated exposure control unit 14 may for instanceimplement the determination of one or more exposure times based onhistograms and cost functions. To this end, dedicated exposure controlunit 14 may for instance receive image data from image sensor 160 viacentral processor 10 to be able to determine a histogram of the lightreception rates with respect to a specific image target, and mayretrieve a plurality of cost functions from processor memory 12 viacentral processor 10. Equally well, unit 14 may receive the image datadirectly from image sensor 160, and/or may information on the costfunction from its own memory. The determined one or more exposure timesmay be forwarded to central processor 10 for controlling shutter unit162. Alternatively, dedicated exposure control unit 14 may controlshutter unit 162 by itself. Dedicated exposure control unit 14 mayfurthermore implement the merging of multiple images of the same imagetarget captured with different exposure times. Dedicated exposurecontrol image 14 may for instance be embodied as a Digital SignalProcessor (DSP), an Application Specific Integrated Circuit (ASIC) or aField Programmable Gate Array (FPGA).

FIG. 2 is a flowchart 200 of a first exemplary embodiment of a methodaccording to the present invention. In this first exemplary embodiment,it is assumed that only one image of an image target is captured, sothat only one exposure time has to be determined.

In a first step 201, a histogram with respect to an image target isdetermined. This step may for instance be performed by central processor10 (see FIG. 1). Each ordinate value of the histogram represents thenumber of image sensor pixels that receive light from the image targetat a light reception rate given as a corresponding abscissa value. Thisordinate value then may be given as an absolute unit (pixels) or as arelative unit related to the overall number of pixels in the imagesensor.

Examples of such a histogram are depicted in dashed lines in FIGS. 4 aand 4 b for a first image target and a second image target,respectively. From the two largest peaks of the histogram of the firstimage target in FIG. 4 a, it can for instance be seen that a lot ofpixels of the image sensor receive light at comparably low lightreception rates, and that there exists a further large amount of pixelsof the image sensor that receive light at comparably high lightreception rates.

Therein, the histograms in FIGS. 4 a and 4 b exemplarily represent thereception of light by the green pixels of the image sensor, since thegreen pixels are activated most frequently and represent most of theluminance. Equally well, light received by red or blue pixels, or thesum or other weighted combinations of the light received by red, greenand blue pixels could be considered when determining the histograms. Theabscissa of the histogram plots of FIGS. 4 a and 4 b is exemplarilygiven in electrons per second. Equally well, it may be possible to usephotons per second, although this measure may be more difficult todetermine and may be demanding in real-time processing. The photons thatare impinging on the image sensor may be filtered by optics, colorfilters, IR filters, so that it is less complex to consider theelectrons collected from the impinging photons by the image sensorpixels. Therein, the quantum efficiency of the image sensor defines howmany electrons can be retrieved from the impinging photons. Thecollected electrons may for instance be converted into digital values byfloating diffusion and analog-to-digital conversion, and by consideringthe exposure time that was used for the determination of the histogram,the full well capacity of the image sensor and the dynamic range of theanalog-to-digital converter, a corresponding light reception rate inelectrons per second may be calculated. Therein, dark current mayalready be removed in the image sensor.

The histograms in FIGS. 4 a and 4 b may thus for instance be determinedby capturing an image target during a fixed exposure time, and thendetermining, e.g. for a grid of equidistant numbers of electrons, fromhow many pixels of the image sensor the respective number of electronswas retrieved during the exposure time. This number of pixels thenrepresents the ordinate value, and the number of electrons divided bythe exposure time yields the corresponding abscissa value. Therein, ithas to be noted that, if the overall exposure time for the determiningof the histogram is chosen too large, saturation of the histogram mayoccur.

Returning to the flowchart 200 of FIG. 2, after the determination of thehistogram in step 201, an exposure time for the image target isdetermined by selecting a cost function representation out of aplurality of cost function representations based on the determinedhistogram. Therein, each cost function representation out of theplurality of cost function representations expresses a performance ofthe image sensor as a function of the light reception rate for aspecific exposure time.

Examples of such cost function representations for exposure times of 9,18 and 36 ms are depicted in FIGS. 4 a and 4 b in solid lines. Fromthese figures, it can readily be seen that the cost functionrepresentation for an exposure time of 9 ms indicates highestperformance of the image sensor in the large light reception rateregime, and that the cost function representation for an exposure timeof 36 ms indicates highest performance of the image sensor in the smalllight reception rate regime.

Cost function representations can be derived based on models of the costfunction of the image sensor, or may be determined by measurements.Furthermore, generation of cost function representations by means ofinterpolation and extrapolation of already available cost functionrepresentations is possible.

The performance of the image sensor expressed by the cost function mayfor instance be a signal-to-noise ratio. The form of the cost functionmay depend, inter alia, on the full well capacity and the quantumefficiency of the image sensor, as well as on noise characteristics. Forinstance, the right part of the cost function may mostly be defined bythe full well capacity and the quantum efficiency of the image sensor,wherein the abrupt decline at the rightmost border of the cost functionmay occur due to saturation. The left part of the cost function maydecline because, in this comparably low light reception rate regime,only few photons may be received by the pixels of the image sensor andthe amount of noise (e.g. due to a noise floor) may be more dominant.

The cost function may furthermore depend on the analog gain that is usedduring the capturing of images. Thus the cost function may depend onboth the exposure time and the analog gain, and instead of onlydetermining an exposure time based on the histogram and the costfunction, it may equally well be possible to determine both an exposuretime and an analog gain for capture of the image target based on thehistogram and the cost function. Some sensors can also utilize digitalgain. Digital gain may be used similarly like analog gain in theanalysis.

Cost function representations for a specific exposure time may forinstance be determined by measuring the full well capacity of the imagesensor, yielding the right end of the cost function representation inthe given exposure time. Then the noise floor may be measured (forinstance by measuring how many electrons are output by the image sensoralthough there is no exposure), yielding the left end of the costfunction representation. Photon shot noise (i.e. the square root oflight electrons) and dark current shot noise (i.e. the square root ofdark current electrons depending on the exposure time) may also be takeninto account.

The plurality of cost function representations according to step 202 offlowchart 200 may for instance be stored in apparatus 1, for instance inprocessor memory 12 or in a memory of dedicated exposure control unit14. The cost function representations may be constant; it may however beadvantageous to modify or weight the cost function representationsbefore determining an exposure time based on the cost functionrepresentations.

For instance, to account for motion of the image target and/or theapparatus 1 (see FIG. 1), it may be advantageous to measure the motionof the image target and/or the apparatus 1, which measurement may forinstance be accomplished by motion detector 161 of apparatus 1 or bycentral processor 10 (e.g. via analysis of differences betweensubsequently captured images). In dependence on the amount of measuredmotion, the cost function representations may then be weighted withfactors, e.g. ranging from 0 to 1.0, wherein cost functionrepresentations for small exposure times may be multiplied with largerweighting factors and cost function representations for large exposuretimes may be multiplied with smaller weighting factors.

An exemplary example of a choice of weighting factors in dependence onthe measured amount of motion and the specific exposure time of the costfunction representation is given in the following table:

Specific Exposure Amount of motion time (ms) Weighting factor No motion6 1.00 No motion 36 1.00 Average motion 6 0.98 Average motion 36 0.70High motion 6 0.95 High motion 36 0.40

The weighting of the cost function representations is advantageouslyperformed prior to or during the determining of the exposure time basedon the histogram and the cost function representations.

In step 202 of flowchart 200, a cost function representation is selectedout of a plurality of cost function representations, wherein theselection is based on the determined histogram for the image target. Abasic rule for the selection of the cost function representation may forinstance be that a cost function representation is selected that revealsgood performance of the image sensor in light reception rate regimeswhere the histogram has large ordinate values. Since each cost functionrepresentation is associated with a specific exposure time, by selectinga cost function representation, also a specific exposure time isselected. For this specific exposure time, it is then guaranteed thatfor a large amount of pixels of the image sensor (as indicated by theordinate values of the histogram), an exposure time is chosen that leadsto a high performance of the image sensor.

A straightforward way to implement the selection of the cost functionrepresentation out of the plurality of cost functions is to determine,for each cost function representation, and for each abscissa value, theproduct between the cost function representation and the histogram andto sum the resulting product values to obtain a sum value. The largestsum value of all cost function representations then identifies the costfunction representation (and the associated exposure time) that isoptimally suited for the histogram.

For example, for the histogram of the first image target in FIG. 4 a andthe histogram of the second image target in FIG. 4 b, applying theabove-described selection approach reveals that selection of the costfunction representation for an exposure time of 9 ms is a good choice.Of course, the cost function representations for exposure times of 36 msand 18 ms can show the smaller light reception rates better (cf. the bigpeaks of the histogram), but if only one exposure time can be used it isadvantageous to use an exposure time of 9 ms for avoiding a huge amountof saturated pixels.

It should be noted that there exist a plurality of further techniques ofselecting the cost function representation from the plurality of costfunction representations.

In a final step 203 of flowchart 200, an image of the image target iscaptured with the exposure time that is associated with the costfunction representation that was selected in step 202. Image capture mayfor instance be triggered by central processor 10 of apparatus 1 (seeFIG. 1) by controlling shutter unit 162 and image sensor 160 of cameraunit 16 accordingly. Equally well, image capture may be triggered bydedicated exposure control unit 14.

After the capture of the image in step 203 of flowchart 200, thecaptured image may for instance be stored in image memory 13 ofapparatus 1 (see FIG. 1), and may optionally be displayed via display11.

In the exemplary embodiment described with reference to the flowchart200 of FIG. 2, it was exemplarily assumed that a plurality of costfunction representations is available, from which an optimal costfunction representation is selected based on the determined histogram.According to the present invention, it may equally well be the case thatonly one cost function representation for one specific exposure time isavailable, and that the exposure time is then determined based on thesingle cost function representation only. This may for instance compriseextrapolation or estimation of cost function representations forspecific exposure times for which no cost function representations areavailable.

For instance, in the scenario of FIG. 4 a, if only the cost functionrepresentation for 6 ms is available (not shown in FIG. 4 a), it may bedetermined, based on the histogram for the first image target and thissingle cost function representation, that a larger exposure time (e.g. 9ms) than 6 ms has to be applied to adequately capture the first imagetarget, since the peak of the cost function representation is known tobe shifted towards smaller light reception range regimes with increasingexposure times. Therein, it may be advantageous if a cost functionrepresentation of a comparably small exposure time (e.g. 6 ms) isavailable, so that the amount of saturated pixels in the measured imageis small. Based on a cost function representation with a comparablyshort exposure time (and corresponding low degree of pixel saturation),cost function representations for larger exposure times may thus bepredicted.

It is also possible that the exposure time is determined based on thedetermined histogram and an analytical model of the cost function, forinstance by extracting a range of light reception rates from thehistogram with high associated histogram ordinate values andcalculating, based on the analytical model of the cost function, forwhich specific exposure time the cost function has high performance inthis range of light reception rates.

FIG. 3 illustrates a flowchart 300 of a further exemplary embodiment ofa method according to the present invention. In this further exemplaryembodiment, it is exemplarily assumed that two images are to becaptured, and that the two images are merged in order to increase thedynamic range as compared to the single capture case. Furthermore, it isassumed that a plurality of cost function representations is available,from which two cost function representations and thus two associatedexposure times are selected, which exposure times are then used in thecapture of the two images of the image target. It is readily understoodby a person skilled in the art that, according to the present invention,it is equally well possible to determine exposure times for more thantwo images and to accordingly capture more than two images. It can alsobe understood by a person skilled in the art that one image can also becaptured by using two or even more exposure times and so the ADconversion may cause a known nonlinear response for all or a group ofpixels.

In a first step 301, a histogram with respect to the image target thatis to be captured is determined. This step is completely analog to step201 of flowchart 200 of FIG. 2, so that the description of step 201 isreadily applicable to step 301 of flowchart 300.

In a second step 302, first and second exposure times are determinedbased on the histogram determined in step 301 and on the cost function.

The benefit of multiple capture with different exposure times is readilyvisible from the exemplary histograms and cost function representationsin FIGS. 4 a and 4 b. For instance, capturing the first image target inFIG. 4 a with an exposure time of 36 ms only, ignores that there existsa significant amount of pixels that receive light at high receptionrates (see the peak in the right part of the histogram for the firstimage target). Light with high light reception rates leads to a quicksaturation of the corresponding pixels of the image sensor, so that,when only capturing a single image with an exposure time of 36 ms, nobright detail will be available from these pixels. In contrast, whenadditionally capturing a second image of the image target with a shortexposure time of 9 ms, the corresponding cost function representation ofthe image sensor has its peak shifted rightwards, so that also highlight reception rates can be adequately covered. The pixels of the imagesensor that receive the light at high light reception rates than will,due to the short exposure time, also provide sufficient detail in thebright part, since saturation of pixels is prevented.

A similar observation holds for the histogram of the second image targetin FIG. 4 b, where the cost function representations for 9 and 18 mswill produce best results.

The first and second exposure times can for instance be determined byselecting, from a plurality of cost function representations (in case ofFIGS. 4 a and 4 b three cost function representations are available),first and second cost function representations that are considered toyield optimum increase of dynamic range.

This may be accomplished in a plurality of ways. For instance, it may beadvantageous to select, from the plurality of cost functionrepresentations, a combination of two cost function representations sothat a sum value is maximized, wherein the sum value for eachcombination is obtained by:

-   -   applying a maximum-value function to the two cost function        representations to obtain a maximum-value cost function        representation, i.e. selecting, for each abscissa value of the        two cost function representations, the larger one of the two        associated cost function representation values (e.g. when        performing this with the cost function representations for 9 and        36 ms in FIG. 4 a, the resulting maximum-value cost function        representation would be equal to the cost function        representation for 36 ms in the low light reception range regime        and, starting from the intersection of both cost function        representations, would be equal to the cost function        representation for 9 ms in the higher light reception range        regime),    -   multiplying, for each light reception rate in a range of        available light reception rates (e.g. the complete abscissa in        FIG. 4 a), the histogram value associated with the light        reception rate and the maximum-value cost function        representation value associated with the light reception rate,        and    -   summing the products obtained for each light reception rate in        the considered range of light reception rates.

Thus by forming a plurality of different combinations of the availablecost function representations (or also of interpolated versionsthereof), and by identifying the maximum combination-specific sum value,the best combination of cost function representations for the histogramand thus the best two exposure times for the capture of the image targetcan be determined.

An alternative approach for determining the first and second exposuretimes for two-fold capture of the image target will be described asfollows.

For instance, an iterative method may be used to determine the first andsecond exposure times. In a first round, some first exposure time isused, which is preferably a small exposure time so that thecorresponding cost function representation covers most of the area ofthe histogram (since the exposure time is small, there may not be somany saturated pixels). Then the cost function representation for thesecond exposure time is searched. This may for instance be accomplishedas described above, i.e. by forming a maximum-value cost functionrepresentation of the already determined cost function representationfor the first exposure time and cost function representation candidatesfor the second exposure time and checking what sum value thismaximum-value cost function representation yields when element-wisemultiplied with the histogram and accordingly summed over the abscissa.

The search of the exposure times should be started by using full dynamic(a very short exposure time) for the first exposure time and by findingas long exposure time that is possible with the available sensor for thenext exposure time.

When the cost function representation with full dynamic has been found,wherein full dynamic means that there are no saturated pixels, it canthen be improved by increasing the used short exposure time (t1) as muchas possible while still not causing saturation. The point (the shortestexposure time) where saturation starts may for instance be defined fromthe determined (e.g. measured) histogram values in the following way:Divide the rate value that utilizes full well capacity in the costfunction representation of the current exposure time by the brightestrate value in the histogram and multiply the result by the currentexposure time. For instance, in FIG. 4 a, when the current exposure timeis 9 ms, the rate value based on the current exposure time (9 ms)utilizing full well capacity is the rate value where the cost functionrepresentation ends (i.e. the right end of the solid line). Thebrightest rate value is the value where the histogram ends (i.e. theright end of the dashed line). So the point (=shortest exposure time)where saturation starts is roughly given as 1.04×9 ms=9.36 ms.

Then the cost function representation for that exposure time iscalculated based on the one or more available (e.g. measured) costfunction representations or interpolations/extrapolations thereof. Thenthe second exposure value is tried to estimate (only a set of availableexposure times is possible, e.g. 2×t1, 4×t1, 8×t1) based on theavailable information so that the sum value of the combination of thetwo cost function representations is maximized. Therein, the optimalsecond exposure time may often be dependent on the area of the histogramwhere the largest amount of pixel values are available. This means thatthe peak in the histogram may often define which exposure should be used(slightly shorter exposure time than the position of peak): E.g. in FIG.4 a, the histogram peak just before the cost function representation for36 ms and in FIG. 4 b the histogram peak just before the cost functionrepresentation for 18 ms exposure time. This may be due to the fact thatthe right end of the cost function is determined by the exposure timeand the full well capacity. But because the form of histogram can bearbitrary then the cost function representation effectively finds thebest combination.

In the above-described approach, it is exploited that there is a set ofcost function representations available and that the cost functionrepresentation for other exposure times may be interpolated based on theavailable information. Equally well, the cost function representationsmay be fully modelled (estimated) by mathematical function if it ispossible.

In steps 303 and 304, then a first image and a second image of the imagetarget are captured with respective exposure times as determined (e.g.via the selection of the two cost function representations) in step 302.

In a step 305, then image data of the first and second image is mergedinto a third image to increase the dynamic range as compared to thesingle capture case. This may for instance be performed on apixel-per-pixel basis, but also for groups of pixels. The rationalebehind this approach is that pixels from the image with shorter exposure(which are likely to reveal more detail in bright image parts) andpixels from the image with larger exposure (which are likely to revealmore detail in dark image parts) are combined into a high-dynamic outputimage.

Two exemplary approaches of merging will be outlined below.

A first approach can be described by the following rule:

if (x ₂<max−thr₁) then x=x₂

else x=x ₁ ·t ₂ /t ₁

Therein, x₁ denotes the pixel value in the first image captured with anexposure time t₁, x₂ denotes the pixel value in the second imagecaptured with an exposure time t₂ (t₂>t₁), x denotes the merged pixelvalue, max is the maximum value that the analog-to-digital converter inthe image sensor can represent, and thr₁ is a security margin that isused to prevent cases when x₂ is not full although is should be (e.g.due to nonlinear image sensor response).

Thus when the dynamic of the analog-to-digital converted is sufficientto represent x₂ (from the longer exposed image), x₂ is used, andotherwise a scaled version of x₁ (from the shorter exposed image) isused, wherein the scaling factor t₂/t₁ indicates that the increase indynamic range in bits that can be achieved by merging the two images islog₂(t₂/t₁). Thus if the second exposure time t₂ is twice the firstexposure time t₁, an increase in dynamic range of 1 bit is achieved.

A second approach of merging can be expressed as follows:

if abs(y ₁ −y ₂)<thr₂ then x=y₂

else x=max(y ₁ ,y ₂)

Therein, in addition to the notation of the previous approach,y₁=x₁·t₂/t₁ and y₂=x₂ hold, abs ( ) returns the absolute value of itsargument, max( ) returns the larger one of its two arguments, and thr₂is a security margin so that the more accurate number is used.

Thus in case of insignificant difference between y₁ and y₂, y₂ is used,and otherwise the maximum value of y₁ and y₂ is used, where y₂ is thescaled version of x₁ with a scaling factor of t₂/t₁, so that once againan increase in dynamic range in bits of log₂(t₂/t₁) can be achieved. Inthe above examples, t2 represents the longer exposure time and t1represents the shorter exposure time.

Nevertheless, the actual order of exposures can be t1, t2 or t2, t1.

Thus when merging the images captured with exposure times of 9 ms and 36ms, respectively (see FIG. 4 a), an increase in dynamic range of 2 bitis achieved, and by merging the images captured with exposure times of 9ms and 18 ms, respectively (see FIG. 4 b), an increase in dynamic rangeof 1 bit is achieved.

After the merging of the image data in step 305 of flowchart 300, themerged image may for instance be stored in image memory 13 of apparatus1 (see FIG. 1), and may optionally be displayed via display 11.

While there have been shown and described and pointed out fundamentalnovel features of the invention as applied to preferred embodimentsthereof, it will be understood that various omissions and substitutionsand changes in the form and details of the devices and methods describedmay be made by those skilled in the art without departing from thespirit of the invention. For example, it is expressly intended that allcombinations of those elements and/or method steps which performsubstantially the same function in substantially the same way to achievethe same results are within the scope of the invention. Moreover, itshould be recognized that structures and/or elements and/or method stepsshown and/or described in connection with any disclosed form orembodiment of the invention may be incorporated in any other disclosedor described or suggested form or embodiment as a general matter ofdesign choice.

Furthermore, it should be noted that the methods disclosed in thecontext of the present invention may be implemented in a variety ofhardware realizations, as for instance in a camera module, an externalaccelerator, an imaging engine, an application processor, a basebandprocessor, to name but a few.

1. A method, comprising: determining a histogram of the number of imagesensor area elements of an image sensor that receive light at specificlight reception rates from an image target; and determining at least oneexposure time for capture of said image target based on said histogramand on a cost function that expresses a performance of said image sensoras a function of light reception rate per image sensor area element andexposure time. 2-46. (canceled)
 47. A computer-readable medium having acomputer program stored thereon, the computer program comprising:instructions operable to cause a processor to determine a histogram ofthe number of image sensor area elements of an image sensor that receivelight at specific light reception rates from an image target; andinstructions operable to cause a processor to determine at least oneexposure time for capture of said image target based on said histogramand on a cost function that expresses a performance of said image sensoras a function of light reception rate per image sensor area element andexposure time.
 48. An apparatus, comprising: a processing unitconfigured to determine a histogram of the number of image sensor areaelements of an image sensor that receive light at specific lightreception rates from an image target; and to determine at least oneexposure time for capture of said image target based on said histogramand on a cost function that expresses a performance of said image sensoras a function of light reception rate per image sensor area element andexposure time.
 49. The apparatus according to claim 3, wherein saidprocessing unit is configured to determine said at least one exposuretime based on said histogram and on a plurality of cost functionrepresentations, wherein each of said cost function representationsexpresses a performance of said image sensor as a function of lightreception rate per image sensor area element for a specific exposuretime.
 50. The apparatus according to claim 4, wherein at least one ofsaid cost function representations is based on measurements or on ananalytical model of said cost function or is obtained from interpolationor extrapolation of other cost function representations.
 51. Theapparatus according to claim 4, wherein said processing unit isconfigured to determine only one exposure time, and wherein saidprocessing unit is configured to determine said one exposure time byselecting, based on said histogram, a cost function representation outof said plurality of cost function representations and by determiningsaid specific exposure time of said selected cost functionrepresentation as said one exposure time.
 52. The apparatus according toclaim 6, wherein said processing unit is configured to select said costfunction representation by multiplying, for each cost functionrepresentation in said plurality of cost function representations, andfor a range of light reception rates, the respectively associatedhistogram value and the respectively associated value of said costfunction representation and summing up the resulting multiplicationproducts to obtain a sum value; by comparing said sum values of all costfunction representations in said set of cost functions to identify thelargest sum value, and by selecting the cost function representationthat produced said largest sum value.
 53. The apparatus according toclaim 3, wherein said processing unit is configured to determine firstand second exposure times for respective first and second captures ofsaid image target.
 54. The apparatus according to claim 8, furthercomprising a camera unit configured to capture said image target withsaid first exposure time to obtain a first image and to capture saidimage target with said second exposure time to obtain a second image;and wherein said processing unit is further configured to merge imagedata of said first and second images into a third image.
 55. Theapparatus according to claim 9, wherein said camera unit is configuredto capture said first image and said second image without temporaloverlap or with temporal overlap.
 56. The apparatus according to claim9, wherein said processing unit is configured to merge said image dataof said first and second image by using, in case of image data of saidsecond image falling below a pre-defined threshold, said image data ofsaid second image as image data for said third image; and by using, incase of image data of said second image equaling or exceeding saidpre-defined threshold, a scaled representation of image data of saidfirst image as image data for said third image, wherein the ratiobetween said second exposure time and said first exposure time is usedas scaling factor.
 57. The apparatus according to claim 9, wherein saidprocessing unit is configured to merge said image data of said first andsecond image by using, in case of a difference between image data ofsaid second image and a scaled representation of image data of saidfirst image falling below a pre-defined threshold, said image data ofsaid second image as image data for said third image, wherein the ratiobetween said second exposure time and said first exposure time is usedas scaling factor; and by using, in case of said difference between saidimage data of said second image and said scaled representation of saidimage data of said first image equaling or exceeding said pre-definedthreshold, the maximum value of said image data of said second image andsaid scaled representation of said image data of said first image asimage data for said third image.
 58. The apparatus according to claim 3,wherein said processing unit is configured to determine first and secondexposure times for respective first and second captures of said imagetarget, and wherein said processing unit is configured to determine,based on said histogram and said cost function, one of said first andsecond exposure times as the largest possible exposure time that stilldoes not cause pixel saturation.
 59. The apparatus according to claim 4,wherein said processing unit is configured to determine first and secondexposure times for respective first and second captures of said imagetarget by selecting, based on said histogram, first and second costfunction representations out of said plurality of cost functionrepresentations and by determining said specific exposure times of saidselected first and second cost function representations as said firstand second exposure times.
 60. The apparatus according to claim 14,wherein said processing unit is configured to select said first andsecond cost function representations by forming a plurality of differentcombinations of two cost function representations out of said pluralityof cost function representations; determining, for each combination ofcost function representations, a sum value by applying a maximum-valuefunction to the two cost function representations to obtain amaximum-value cost function representation, by multiplying, for a rangeof light reception rates, the respectively associated histogram valueand the respectively associated maximum-value cost functionrepresentation value and by summing the resulting products; andselecting the two cost function representations of that combination ofcost function representations that yields the largest sum value.
 61. Theapparatus according to claim 4, further comprising a motion detectorconfigured to detect motion of at least one of said image target andsaid image sensor; wherein said processing unit is configured to weightsaid cost function representations in dependence on said detected motionprior to or during said determining of said at least one exposure time.62. The apparatus according to claim 3, wherein said performance of saidimage sensor is related to a signal-to-noise ratio of said image sensor.63. The apparatus according to claim 3, wherein said cost functionexpresses a performance of said image sensor as a function of lightreception rate per image sensor area element, exposure time and analoggain of said image sensor, and wherein said processing unit isconfigured to determine at least one exposure time and at least oneanalog gain for capture of said image target based on said histogram andon said cost function.
 64. The apparatus according to claim 3, whereinsaid apparatus is a digital camera or an electronic device that isequipped with a digital camera.
 65. The apparatus according to claim 3,wherein said apparatus is a module for a digital camera or for anelectronic device that is equipped with a digital camera.